Skip to content

[testing] Tests mark themselves UNSUPPORTED if features aren't available#35

Merged
rolfmorel merged 7 commits intomainfrom
users/rolfmorel/lit-cleanup
Dec 16, 2025
Merged

[testing] Tests mark themselves UNSUPPORTED if features aren't available#35
rolfmorel merged 7 commits intomainfrom
users/rolfmorel/lit-cleanup

Conversation

@rolfmorel
Copy link
Contributor

@rolfmorel rolfmorel commented Dec 9, 2025 •
edited
Loading

Also loosens dependency upon uv for running tests -- all that is required is that lit is running in a suitable environment (e.g. through uv run lit . or by first entering an venv and then lit .).

Also splits lighthouse.utils into two so that the examples/tests that don't actually depend on torch accidently end up with that dependency (there was one such such instance already).

@rolfmorel
Copy link
Contributor Author

rolfmorel commented Dec 9, 2025 •
edited
Loading

E.g. if you didn't install torch dependencies (e.g. did uv sync without --extra ingress-torch-cpu flag), then we get:

$ lit .
-- Testing: 8 tests, 8 workers --
UNSUPPORTED: Lighthouse test suite :: examples/ingress/torch/mlp_from_model.py (1 of 8)
UNSUPPORTED: Lighthouse test suite :: examples/ingress/torch/mlp_from_file.py (2 of 8)
UNSUPPORTED: Lighthouse test suite :: examples/xegpu_matmul/matmul.py (3 of 8)
UNSUPPORTED: Lighthouse test suite :: examples/mlir/compile_and_run.py (4 of 8)
UNSUPPORTED: Lighthouse test suite :: examples/ingress/convert-kernel-bench-to-mlir.py (5 of 8)
UNSUPPORTED: Lighthouse test suite :: examples/llama/test_llama3.py (6 of 8)
PASS: Lighthouse test suite :: examples/schedule/transform_a_payload_according_to_a_schedule.py (7 of 8)
PASS: Lighthouse test suite :: examples/ingress/mlir_gen/generate-linalg-3layer-mlp.sh (8 of 8)

And if you then install torch but still forgot to/didn't fetch the KernelBench third_party repo:

$ lit .
-- Testing: 8 tests, 8 workers --
UNSUPPORTED: Lighthouse test suite :: examples/ingress/convert-kernel-bench-to-mlir.py (1 of 8)
PASS: Lighthouse test suite :: examples/schedule/transform_a_payload_according_to_a_schedule.py (2 of 8)
PASS: Lighthouse test suite :: examples/ingress/mlir_gen/generate-linalg-3layer-mlp.sh (3 of 8)
PASS: Lighthouse test suite :: examples/mlir/compile_and_run.py (4 of 8)
PASS: Lighthouse test suite :: examples/llama/test_llama3.py (5 of 8)
PASS: Lighthouse test suite :: examples/xegpu_matmul/matmul.py (6 of 8)
PASS: Lighthouse test suite :: examples/ingress/torch/mlp_from_file.py (7 of 8)
PASS: Lighthouse test suite :: examples/ingress/torch/mlp_from_model.py (8 of 8)

Without this change the above UNSUPPORTED tests would show FAILED instead.

adam-smnk
adam-smnk approved these changes Dec 9, 2025
View reviewed changes
tkarna
tkarna approved these changes Dec 9, 2025
View reviewed changes
Copy link
Contributor

@tkarna tkarna left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. At the moment we are only testing the "everything is installed" case in CI, right?

@rengolin
Copy link
Member

rengolin commented Dec 9, 2025

why is the kernelbench still unsupported after installing torch?

@rolfmorel
Copy link
Contributor Author

rolfmorel commented Dec 9, 2025

why is the kernelbench still unsupported after installing torch?

The example speaks for itself 😉

$ uv run examples/ingress/convert-kernel-bench-to-mlir.py
ERROR: KernelBench repo not found.
NOTE: Pull in dependency with: git submodule update --init third_party/KernelBench

@rolfmorel rolfmorel force-pushed the users/rolfmorel/lit-cleanup branch from e5dd205 to efff1b1 Compare December 9, 2025 18:53
@rolfmorel
Copy link
Contributor Author

rolfmorel commented Dec 9, 2025

At the moment we are only testing the "everything is installed" case in CI, right?

Indeed. If people want this UNSUPPORTED mechanism tested in CI as well, do let me know.

@rolfmorel rolfmorel force-pushed the users/rolfmorel/lit-cleanup branch 2 times, most recently from b58358f to 84fd5fb Compare December 15, 2025 21:33
@rolfmorel
[testing] Tests mark themselves unsupported if features aren't available
82aeef1 
Also loosens dependency upon `uv` for running tests -- all that is
required is that `lit` is running in a suitable environemnt (e.g.
through `uv run lit .` or by first entering an venv and then `lit .`).
@rolfmorel
Split out torch dependent functionality in lighthouse.utils
e91ed21 
Means at least one more example can run without torch being installed.
@rolfmorel
Fixes for recently merged workload component
dd87ca4 
1# with '#' will be ignored, and an empty message aborts the commit.
@rolfmorel
Fewer kernelbench examples to speed it up further
ff999ec
@rolfmorel
REQUIRES in cfg instead
ee64cf3
@rolfmorel
Revert to %PYTHON in RUN lines
f095c8d
@rolfmorel rolfmorel force-pushed the users/rolfmorel/lit-cleanup branch from 84fd5fb to 64bf0ab Compare December 15, 2025 21:41
@rolfmorel rolfmorel force-pushed the users/rolfmorel/lit-cleanup branch from 64bf0ab to e13be22 Compare December 15, 2025 21:53
@rolfmorel rolfmorel merged commit 8922c63 into main Dec 16, 2025
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@adam-smnk adam-smnk adam-smnk approved these changes

@tkarna tkarna tkarna approved these changes

@rengolin rengolin Awaiting requested review from rengolin

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@rolfmorel @rengolin @tkarna @adam-smnk